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Abstract 


This paper describes the development of a custom built Digital Signal Processing (DSP) 
printed circuit board designed to implement the Advanced Real Time Vibration 
Monitoring Subsystem proposed by Marshall Space Flight Center (MSFC) 

Transportation Directorate 2 in 2000 for the Space Shuttle Main Engine Advanced Health 
Management System (AHMS). This Real Time Vibration Monitoring System (RTVMS) 
is being developed for ground use as part of the AHMS Health Management Computer - 
Integrated Rack Assembly (HMC-IRA). The HMC-IRA RTVMS design contains five 
DSPs which are highly interconnected through individual communication ports, shared 
memory, and a unique communication router that allows all the DSPs to receive digitized 
data from two multi-channel analog boards simultaneously. This paper will briefly cover 
the overall board design but will focus primarily on the state-of-the-art simulation 
environment within which this board was developed. This 16-layer board with over 1800 
components and an additional mezzanine card has been an extremely challenging design. 
Utilization of a Mentor Graphics simulation environment provided the unique board and 
system level simulation capability to ascertain any timing or functional concerns before 
production. By combining VHDL, Synopsys Software and Hardware Models, and the 
Mentor Design Capture Environment, multiple simulations were developed to verify the 
RTVMS design. This multi-level simulation allowed the designers to achieve complete 
operability without error the first time the RTVMS printed circuit board was powered. 
The HMC-IRA design has completed all engineering and deliverable unit testing. 


1. Introduction 

Traditional design verification methods have primarily consisted of an emphasis on the 
initial cut of a printed circuit board. The initial hardware is utilized as a test unit where 
the debugging of the integrated design begins. The FPGA functionality can be verified 
independently; however, its operation in the overall design cannot. With this design 
method, components must be procured immediately, hardware and software designs are 
locked into initial decisions, and modifications tend to be inefficient. Multi-level 
simulations provide the capability to verify at the component (FPGA), board, and system 
level before any hardware is procured. The emphasis of the design is on the simulations 
where the functionality and timing of any design level can be verified. More time is spent 
in the design phase, and any modifications are easily implemented within the simulation 
environment. A flow chart of the design methodologies is shown in Figure 1 . 
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Figure 1 : Design Methodology 

The Real Time Vibration Monitoring System design of the Health Monitoring 
Component Integrated Rack Assembly utilized the simulation based methodology for 
design and verification. The following paper focuses on the multi-level simulation 
environment, and how it was utilized to verify the operation of the VME and MGBC 
FPGAs, the RTVMS board, and the integration of the RTVMS in the HMC-IRA system. 


2. Health Management System (HMC-IRA) Overview 


A Real-Time Vibration Monitoring System (RTVMS) was developed in 1996 by MSFC 
and currently operates at Stennis Space Center during Space Shuttle Main Engine 
(SSME) test firings. The RTVMS provides real-time vibration analysis and health 
monitoring capabilities during engine operation by producing vibration spectral data from 
critical SSME components. This vibration analysis provides the capability to activate a 
vibration flight redline for engine high-pressure turbo machinery. In early 2000, the 
MSFC Shuttle Main Engine Project Office decided to pursue implementation of the 
RTVMS technology for Space Shuttle Flights. This advanced RTVMS would contain all 
the features of the current SSC ground RTVMS system, but would also include additional 
algorithms that would also evaluate the vibration data in the phase domain 2 . 

The first step toward implementing the flight RTVMS systems was to develop a ‘flight- 
like’ health management system that incorporated the RTVMS and other existing engine 
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monitoring systems. This system is called the Health Management Computer - 
Integrated Rack Assembly (HMC-IRA). The ground based HMC-ERA simulates an 
Advanced Health Management System (AHMS) Space Shuttle Main Engine Controller 
(SSMEC). It provides a RTVMS design that can transition to a flight system design with 
only form-fit modifications. The HMC-IRA also provides interfaces for the AHMS 
SSMEC, SSME Sensors, and the Stennis Space Center Data Acquisition System. 

The HMC-IRA consists of a total of 7 boards that include a combination of custom and 
COTS designs that are integrated by a custom VME backplane as shown in Figure 2. 

The HMC-IRA operates by receiving analog accelerometer inputs from various locations 
on the Space Shuttle Main Engine. Two Engine Analog Data Interface (EADIF) cards in 
the HMC-IRA sample the analog signals, digitize the data and send it to each RTVMS 
card thru independent high-speed communication port interfaces. At the RTVMS, 
complex algorithms are executed on the data and results are stored locally for the System 
Controller to access. The raw data is stored to the 1 .5 Gbyte SEAKR Engineering 
memory card over the VME backplane by the RTVMS cards. 


HMC-IRA Data Flow 



High Speed 
Serial 

Communication 


SYSTEM 

LINEAR 

CONTROLLER 

ENGINE 

PowerPC 

MODEL 

(LEM) 

PowerPC 


SEAKR 
1.5GByte 
Non -Volatile 
Flash 
Memory 


Phase 1 
SSME Block II 
Controller 


VME Backplane 





Gimbal& 


Conditioned 


Turbine 


Accelerometer 


SYSTEM 2 
(RTVMS -2) 


ENGINE 

ANALOG 

DATA 

INTERFACE B 
(EADIF-B) 


C40 

Master 


C40 

Master 


REAL-TIME 

VIBRATION 


REAL-TIME 

VIBRATION 


MONITORING MONITORING 


SYSTEM 1 
(RTVMS-1) 


ENGINE 

ANALOG 

DATA 

INTERFACE A 
(EADIF-A) 


Raw Data storage to Memory, LEM Data Transfers, System Controller Transfers 

Figure 2: HMC-IRA 

HMC-IRA system development and verification is complete for the three deliverable 
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units. One deliverd HMC-IRA unit has been integrated in the Boeing Rocketdyne 
Controller Simulation Lab and is supporting software development. The other two 
deliverables are waiting for integration into systems at Stennis Space Center and the 
MSFC SSMEC Hardware Simulation Lab. 


3. HMC-IRA RTVMS Design Overview 

The RTVMS was the first EI31 multiple/parallel processing DSP design. It was required 
to have a path to flight, but was built with commercial version of flight quality parts. A 
general description of the RTVMS is shown below. 

General Description 

> A32/D32 VME Master w/ Block and RMW transfer capability 

> A24/D32 VME Slave ^ 8K x 32 Dual-Port SRAM 

> 5 Texas Instruments DSPs 

> 2Mbytes of SRAM on Local Bus of each DSP 

> 2Mbytes of SRAM on Global Bus of each DSP 

> 128Kbytes of EEPROM on Global Bus of each DSP for boot operations 

> 32Kbytes Dual-Port SRAM on Global Bus that is shared between the DSPs 

> Communications Port Interface with EADIFA and EADIFB boards 

> Communications Port Interface between the DSPs 

> 2 Actel A54SX32A FPGAs for various logic applications 
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Because of the quantity of parts, the 
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Figure 3: HMC-IRA Real Time Vibration Monitoring System (RTVMS) Card 
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Five TI 320C40 DSPs running at 40MHz were chosen to provide the processing power 
capable of monitoring the discrete turbopump vibration components in real time. Each 
DSP interfaces with local and global buses that are completely separate. The global bus is 
used to access the shared Dual Port SRAM, the 32K x 32 EEPROMs, the 512K x 32 
SRAM and the shared VME Bus as shown in Figure 4. The local bus contains a 5 12K x 
32 SRAM and the Multi Global Bus Controller (MGBC) FPGA registers. Each bus 
contains a STRBO and STRB1 that can be mapped to any section of memory by internal 
DSP registers. Each strobe has support registers that allow the control of such parameters 
as wait states and memory cycle timing. The memory map for the local and global bus is 
shown in Figure 5. 

Any five of the DSPs can access the Dual-Port SRAM (DPSR) or the VME bus thru 
arbitration that is handled in the MGBC FPGA. The MGBC incorporates a round robin 
priority scheme to permit bus access to the VME or DPSR. Once the DSP gains access to 
one of the shared buses, it is at the discretion of the software as when to release it. The 
VME FPGA controls VME bus accesses and timing. The RTVMS Bus Architecture and 
memory maps for the local and global bus are shown below. 



Figure 4: RTVMS DSP Bus Architecture 
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Figure 5: RTVMS DSP Local and Global Memory Map 


Each DSP has 6 communication ports (comports) that are unidirectional. For every DSP, 
comports 1,3, 4, 5 are inputs while 0 and 2 are outputs. Each comport consists of 8 bits of 
data, STRB, and RDY. The RTVMS Comport Architecture is shown in Figure 6. 

The PC Comport Interface provides a communications interface between an external 
peripheral, such as a laptop, and the Master DSP on the RTVMS board. Comports 0 and 
1 of the Master DSP are utilized for this interface when the Comport Mux Select (CMS) 
signal is ‘1’. When the CMS is ‘O’, the PC Comport buffers are disabled and comports 0 
and 1 are used for the Board-to-Board Interface. The CMS is only available to the 
Master DSP. 

The Board-to-Board Comport Interface provides a communication path between the 
RTVMS- 1 and RTVMS-2 boards in the HMC-IRA. The Master DSP of each RTVMS 
board determines which DSP (Master or Slave4) can communicate via the board-to-board 
interface by controlling the Comport Mux Select (CMS) register value. The Master DSP 
is the only one that can access this register. 
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The EADIF Comport Interface provides each DSP with the capability of receiving 
digitized data from the two EADIF boards. Comport 4 is used to receive raw data from 
the EADIF-A board while Comport 5 is used for data from the EADIF-B board. 

The DSP to DSP Comport Interface provides each DSP with the capability to 
communicate with physically adjacent DSPs. Comports 0 and 1 are utilized to 
communicate with the DSP that is before it in the sequence, while comports 2 and 3 
communicate with the DSP that is after it in the sequence. The comports are 
unidirectional. 
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Figure 6: RTMVS Communication Port (Comport) Interfaces 
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4. RTVMS Simulation Environment 


The simulation environment is composed of three critical elements: Hardware 
Description Language (HDL), Synopsys Software Models, and Synopsys Hardware 
Models. The HDL can either be VHDL, Verilog, or a combination, but for this 
simulation only VHDL was used. The VHDL used in the simulation can be behavioral or 
structural as shown in Figure 7. The behavioral VHDL can be simulated directly to verify 
functionality; however, the behavioral VHDL must be run thru a synthesis tool, then 
followed by a vendor specific place and route tool to generate the structural VHDL that 
contains timing information to be used in simulation. Timing information is generated in 
a standard delay file (.sdf) that is utilized by the simulator for the timing simulation. The 
simulator provides the opportunity to vary the delay in the .sdf file. For this design, the 
Leonardo software was used for synthesis, while Actel Designer was used for place and 
route. The VME FPGA controlled VME access and cycles while the MGBC VME 
controlled bus arbitration (DPSR and VME), Watchdog operations, and EADIF Comport 
data transfers thru numerous internal registers for each DSP. 


VHDL 


Behavioral 

(Functional) 

Simulation 


CASE addr IS 

WHEN " 01000 - => flash <= ’O’; 
WHEN -0100V => flash <= V; 
WHEN -01010- => mps_pwr <= V; 
WHEN -01011- => mps_pwr<= ’O'; 
WHEN -01100- => eng_pwr <= 1 ’; 
WHEN "01101 " => eng_pwr <= ’O’; 
WHEN "Oil 10" => x_tvc_en <= 'O’; 
WHEN -01111- => x_tvc_en <= '1 ’; 


VHDL logic implemented in 2 Actel A54SX32A FPGAs 



Figure 7: Behavioral and Structural VHDL 
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The Synopsys software smartmodels are licensed models of components or protocols 
such as memories, logic circuits, VME communication, etc. A simplified example is 
shown in Figure 8. These models are used to verify interfacing properties such as setup 
and hold timing, bus contention, etc. and have some adjustable parameters that can be 
optimized for application. In the HMC-IRA RTVMS design, smartmodels are used to 
simulate the DPSR, SRAM, EEPROM, and other miscellaneous logic (latches, buffers, D 
flip flops, etc). 
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Figure 8: Synopsys Smartmodel 


Hardware models are physical devices built by Synopsys and are interfaced to the 
hardware modeler. The hardware model silicon, inside the hardware models, 
communicates with the simulator thru an Ethernet LAN allowing execution of elementary 
software code attached to memory smartmodels in the schematic. The schematic design 
and simulation tools reside on individual office computers as shown in Figure 9. For the 
RTVMS design, one TMS320C40 hardware model was utilized to run simulations that 
included as many as 10 DSPs. 
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Figure 9: Synopsys Hardware Model Interface 


The elements of simulation are “connected” together in the Hierarchal Mentor Graphics 
Design Capture Environment as shown in Figure 10. In this environment, each box can 
represent a schematic or logical view (VHDL) of the design. Once all the elements have 
been entered into the design capture environment, the tool will generate a top-down 
methodology VHDL netlist with the components connected. This could also be 
accomplished thru hand instantiation, but the graphical environment is easier with larger 
designs. This environment is also where timing attributes are added to smartmodels and 
hardware models for structural simulations. 

For the VHDL, a symbol is generated by the designer with the input and output pins 
added. The view attribute of the symbol must be defined to be VHDL, and a VHDL 
design file must be attached to the symbol. Anytime that this symbol is entered, the 
VHDL design file appears in a text editor. 

For the Synopsys smart models, schematic shells can be created by hand or by the Mentor 
MTISwiftgen utility. The schematic shells are used in the design capture environment to 
represent the smartmodel component. The shells serve as the connection between the 
smartmodel VHDL and the design capture environment by utilizing the shell attributes. 

For the Synopsys hardware models, a similar shell is created to link the model and the 
design environment. System Environment parameters and a VHDL interface generated 
by Synopsys connect the design capture shell to the network hardware model. 
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Figure 10: Mentor Graphics Design Capture Environment 


Once all the elements have been 
combined in the design capture, a 
VHDL netlist is generated for the 
Mentor ModelSim Simulation 
Environment representing all 
components of the design. In this 
environment shown in Figure 11, 
all the VHDL is compiled for 
correctness, and ModelSim 
provides a waveform window to 
observe any chosen signals in the 
design. Once the desired signals 
are entered in the waveform 
window, the simulation is run to 
verify the signals. The timing 
attributes for the FPGAs are 
controlled in this environment. 


Figure 11: Mentor Graphics ModelSim Environment 
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The HMC-IRA RTMVS design flow is shown in Figure 12. This process is repeated 
until the desired results are achieved. The first simulations are run with the behavioral 
FPGA code to verify functionality. After the functionality is verified, the code is 
synthesized to structural VHDL and the simulations are rerun to verify the timing. The 
functional simulations would not include the Synplicity/Leonardo Synthesis Tools block 
since no synthesis is needed. 
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Figure 12: RTVMS Design Flow 
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5. HMC-IRA RTVMS Simulations 


The initial RTVMS simulations consisted of verifying the operation of the TMS320C40 
DSP Hardware Model with its surrounding logic utilizing the DSP Simulation Schematic 
shown in Figure 13. This was accomplished by executing C test code that was stored in 
four 32K x 8 EEPROMS Software Model located in the hierarchal EEPROM block in the 
schematic below. The test code was written and compiled utilizing Texas Instrument’s 
(TI) C40 Code Composer. The compiled code was partitioned into four sections utilizing 
the Hex 30 command that accompanies TI Code Composer. The command is shown 
below. 

Hex 30 EEPROM.out -i -romwidth 8 -memwidth 32 

-o EEPROM.lol -o EEPROM.lo2 -o EEPROM.hil -o EEPROM.hi2 

The four output files shown above were then converted to Intel hex format utilizing a 
MSFC custom conversion utility. Each EEPROM Software Model instance in the 
EEPROM block below contains a memory file attribute that linked the model with a 
corresponding EEPROM Intel hex format file. At boot in the ModelSim Simulation 
Environment, the TMS320C40 would execute the code located in its EEPROM files. 

With this setup in place, it is simple to verify access to the Local and Global SRAM 
Software Models as no other outside logic is needed. Multiple simulations were run 


where the timing attributes on the hardware model and the SRAM software models where 
modified to verify the worst case memory cycles. 

'lijwts'. • 



Figure 13: RTVMS TMS320C40 DSP Simulation Schematic 
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Once the hardware model and its memory were verified, the simulation schematic was 
expanded to include RTVMS board level and FPGA verification. The Master, Slavel 
and Slave2 DSP blocks in Figure 14 include the verified schematic in Figure 13. The 
Mezzanine block contains the Slave 3 and Slave 4 DSP blocks. With this board level 
schematic, multiple simulations were used to verify the following: 

> Dual Port SRAM Access and Dual Port SRAM Arbitration inside MGBC FPGA 

> MGBC FPGA Watchdog Operation 

> MGBC FPGA Registers Access 

> DSP-to-DSP comport communications. 


Once the functionality of the board was verified, the FPGA code was synthesized and the 
simulations repeated multiple times. The timing of the simulations was manipulated by 
changing the timing attributes of the FPGAs, the Hardware Models, or the Synopsys 
software models. 
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Figure 14: RTVMS Board Simulation Schematic 


The top level schematic is shown in Figure 15 for the RTVMS System simulation and 
represents a VME Backplane interconnect. The System Controller, EADIF, RTVMS- 
SN1 and RTVMS-SN2 represent boards in the HMC-IRA VME Backplane. A VME 
Hardware Verification Model represents the System Controller block. This model is 
actually an elegant Synopsys software model that is controlled with PCL code and could 
serve as a VME bus arbiter, VME Interrupt Arbiter, VME Master and VME Slave. The 
Termination Resistors block served as the pull-up resistors on the backplane. The EADIF 
and RTVMS blocks represent the custom board designs of the HMC-IRA and have been 
verified thru simulations independently. The RTVMS-SN1 and RTVMS-SN2 blocks are 
identical and include the RTVMS Board Schematic shown above. With this system level 
schematic, multiple simulations were used to verify the following: 


> RTVMS VME Accesses (Single, Block, Read-Modify-Write) of the System 
Controller 

> Execution of VME Interrupts 

> On-board VME Arbitration 

> VME accesses of the RTVMS DPSR by the System Controller 

> EADIF-A to RTVMS Comport Operations 


Once again, the timing of the simulations was manipulated by changing the timing 
attributes of the FPGAs, the Hardware Models, or the Synopsys software models. 
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Once the simulations were completed, EI3 1 collaborated with the Office of Logic Design 
at Goddard Space Flight Center to provide an independent review of the RTVMS VME 
FPGA. The review provided detailed insight into the functionality and timing of the 
VME FPGA, and resulted in multiple recommendations that were implemented in the 
final RTVMS design. 


6. Summary 

Box and Board Level Simulations provide a unique capability to be able to take an in- 
depth view of the design before any hardware is built. This simulation based design 
methodology has the following advantages: 

> More complex design capability since the functionality and timing of the design 
can be verified at the beginning of the design phase. 

> Lower cost since fewer design iterations are needed 

> Reduced debug time since most of the problems can be discovered during the 
simulations 

> Higher reliability due to verifying the timing of the design 

> The potential to substantially decrease development time over traditional methods 
if the simulations are used to their full capability 

> Test code written for the hardware models can be used as a foundation for the 
development of system software 

> At the system simulation level, multiple designs can be verified simultaneously 
However, there are disadvantages to this design methodology, and they are as follows: 

> Simulations are limited to short time segments that may not allow the designer to 
fully excite the board in a manner that is preferable. Simulations are limited by 
memory on the computer, speed of the computer, bandwidth over the network, 
etc. Simulations run for 500ms can take an hour or longer to complete. 

> Simulation tools are expensive. 

Upfront design time is increased that may result in a longer time to initial printed 
circuit board. 

Items to Consider: 

> Simulation does not equal analysis. In the simulations, signal values change 
immediately with no interference from reflections, signal crosstalk, line 
impedance etc. that can degrade the timing of signals. Simulations do not provide 
enough coverage for critical implementations. For flight programs, analysis 
should be provided for critical signals to provide the appropriate level of 
confidence in the timing of the system. 

> Simulation schematics do not convert directly to build schematics. Although the 
simulation schematics can serve as a base for further schematics, they do not 
translate directly to a printed circuit board layout design. 
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The HMC-IRA RTMVS boards have completed all board level and system level 
verification tests. During the testing for this board, no design errors were encountered for 
the fifteen Engineering Units (EU) and Brassboards. Only slight routing modifications to 
the design were made between the EU and Brassboards. The EU actually met all of the 
RTVMS requirements and a re-spin of the board would not have been required, although 
it was justified by the simplified routing arrangement. Because of the detailed 
simulations and schedule time allotted for performing the simulations, no functional or 
timing errors have occurred. 

2 AIAA-2000-3622, Advanced Engine Health Management Applications of the SSME 
Real-Time Vibration Monitoring System, Tony R. Fiorucci (NASA MSFC), David R. 
Lakin II (NASA MSFC), and Tracy D. Reynolds (Optical Sciences Corporation). 
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Acronyms 


AHMS 

Advanced Health Monitoring System 

CMS 

Comport Mux Select 

DPSR 

Dual-Port SRAM 

DSP 

Digital Signal Processor 

EADEF 

Engine Analog Data Interface 

FPGA 

Field Programmable Gate Array 

HDL 

Hardware Description Language 

HMC-IRA 

Health Monitoring Computer Integrated Rack Assembly 

MGBC 

Multi-Global Bus Controller 

MSFC 

Marshall Space Flight Center 

RTVMS 

Real Time Vibration Monitoring System 

SSME 

Space Shuttle Main Engine 

TI 

Texas Instruments 

VHDL 

VHSIC Hardware Description Language 

VHSIC 

Very High Speed Integrated Circuits 
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